1 . Field of the Invention
The present invention is generally directed to neural networks. More particularly, the present invention is directed to learning in neural networks.
2 . Background Art
Neural networks attempt to achieve autonomous behavior based on a network of simulated neurons that are connected in a manner suggestive of connections between real neurons in humans. In humans, a first neuron may fire in response to an external stimulus. In response to the firing of the first neuron, other neurons connected to the first neuron may also fire.
Similarly, a first simulated neuron in an input layer of a neural network can become active (e.g., fire) in response to stimulus to the neural network. One or more simulated neurons connected to the first simulated neuron may become active (e.g., fire) in response to the activity (e.g., firing) of the first neuron. Whether the activity of the first simulated neuron causes other simulated neurons to become active is dependent on at least two factors: (i) a weight associated with a connection between the first simulated neuron and each other simulated neuron to which the first simulated neuron is connected; and (ii) the threshold activity level required to cause each other simulated neuron to become active.
To train the neural network (i.e., to cause the neural network to behave in a desired way), the weight associated with each connection is adjusted in response to different types of stimuli. A learning rule is the rule applied to determine how the weight changes during each time step.
A well-known learning rule is called the BCM learning rule, after Bienenstock, Cooper, and Munro who introduced this rule in 1982. See Bienenstock, E. L., Cooper, L. N, and Munro, P. W, “Theory for the Development of Neuron Selectivity: Orientation Specificity and Binocular Interaction in Visual Cortex,” Journal of Neuroscience, 2:32-48 (1982), the entirety of which is hereby incorporated by reference herein.
A drawback of the conventional BCM learning rule is that it may cause multiple neurons of a neural network to respond to the same feature of an input signal. As a result, another interesting feature of the input signal may not be responded to by any neurons of the neural network.
Given the foregoing, what is desired are systems, apparatuses, and methods for implementing a neural network that uses an improved learning rule.